System and method for automatic, unsupervised paraphrase generation using a novel framework that learns syntactic construct while retaining semantic meaning

ABSTRACT

A system includes a question answering system executed by a computer, a processor, and a memory coupled to the processor. The memory is encoded with instructions that when executed cause the processor to provide training for training the question answering system. The training system is configured to receive a plurality of bidirectional disjunctive logical forms which include two directional disjunctions of differences between a first logical form of a first sentence and second logical form of a second sentence, realize the plurality of bidirectional disjunctive logical forms to generate a first plurality of paraphrases of the first and second sentence, score each of the first plurality of paraphrases based on textual similarity between the first plurality of paraphrases and the first and second sentences, and prune the first plurality of paraphrases to generate a second plurality of paraphrases based on the scores of each of the first plurality of paraphrases.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT (IFAPPLICABLE)

N/A.

STATEMENT REGARDING PRIOR DISCLOSURES BY THE INVENTOR OR A JOINTINVENTOR (IF APPLICABLE)

N/A.

BACKGROUND

The present invention relates to training of cognitive computingsystems, and more specifically, to techniques and mechanisms forimproving the results generated by a Question and Answer system bygenerating enriched ground truth and providing the enriched ground truthfor training the system.

With the increased usage of computing networks, such as the Internet,users can easily be overwhelmed with the amount of information availablefrom various structured and unstructured sources. However, informationgaps abound as users try to piece together what they believe to berelevant during searches for information on various subjects. To assistwith such searches, research has been directed to creating cognitivesystems such as Question and Answer (QA) systems that take an inputquestion, analyze the question, and return results indicative of themost probable answer or answers to the input question. QA systemsprovide automated mechanisms for searching through large sets of sourcesof content, e.g., electronic documents, and analyze them with regard toan input question to determine an answer to the question and aconfidence measure as to how accurate an answer to the question mightbe.

The quality of the responses provided by a QA system is tied to thetraining provided to the system. When a cognitive system is trained,ground truth is provided to the system. The quality of system training,and in turn, the quality of the cognitive system is determined by thequality of the ground truth used to train the system. Therefore, themore comprehensive the ground truth, the higher the quality of thesystem training.

SUMMARY

According to an embodiment, a method includes receiving, by a surfacerealizer of a paraphrase generation system, a plurality of bidirectionaldisjunctive logical forms. The plurality of bidirectional disjunctivelogical forms includes two directional disjunctions of differencesbetween a first logical form of a first sentence and a second logicalform of a second sentence. An embodiment of the method may also includeconverting, by the paraphrase generation system, the first sentence intothe first logical form by dependency parsing the first sentence toidentify parts of speech of the first sentence thereby generating afirst parsed representation of the first sentence. An embodiment of themethod may also include converting, by the paraphrase generation system,the second sentence into the second logical form by dependency parsingthe second sentence to identify parts of speech of the second sentencethereby generating a second parsed representation of the secondsentence. An embodiment of the method may also include generating, bythe paraphrase generation system, the bidirectional disjunctive form bycomparing the first parsed representation of the first sentence with thesecond parsed representation of the second sentence to determine a wordfrom the first sentence that corresponds with a word from the secondsentence. The method also includes realizing, by the surface realizer,the plurality of bidirectional disjunctive logical forms to generate afirst plurality of paraphrases of the first and second sentences. Themethod also includes scoring, by the paraphrase generation system, eachof the first plurality of paraphrases based on textual similaritybetween the first plurality of paraphrases and the first and secondsentences. In an embodiment, each of the first plurality of paraphrasesis scores by scoring a first paraphrase of the first plurality ofparaphrases higher than a second paraphrase of the first plurality ofparaphrases in response to the first paraphrase having a highersyntactic variation than the second paraphrase. The method also includespruning, by the paraphrase generation system, the first plurality ofparaphrases based on the scores of each of the first plurality ofparaphrases. In an embodiment, the first plurality of paraphrases ispruned by removing a lowest score paraphrase of the first plurality ofparaphrases. The lowest score paraphrase has a score lower than allother paraphrases of the first plurality of paraphrases. An embodimentof the method may also include storing, by the paraphrase generationsystem, the second plurality of paraphrases as ground truth fortraining. An embodiment of the method may also include training thequestion answering system utilizing the ground truth.

In another embodiment, a system/apparatus is provided. Thesystem/apparatus includes a question answering system executed by acomputer, one or more processors, and memory. The memory is encoded withinstructions that when executed cause the one or more processors toprovide a training system for training the question answering system.The training system may be configured to perform various ones of, andvarious combinations of the operations described above with respect toembodiments of a method.

In a further embodiment, a computer program product including a computerreadable storage medium encoded with program instructions is provided.The program instructions are executable by a computer to cause thecomputer to perform various ones of, and various combinations of theoperations described above with respect to embodiments of a method.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an illustrative block diagram of a system that providestraining of a question answering system in accordance with variousembodiments;

FIG. 2 shows an illustrative block diagram of a training system thatincludes a paraphrase generation system to generate paraphrases ofground truth questions to provide training of a question answeringsystem in accordance with various embodiments;

FIG. 3 shows a flow diagram illustrating aspects of operations that maybe performed to generate paraphrases of ground truth questions to traina question answering system in accordance with various embodiments;

FIG. 4 shows a flow diagram illustrating aspects of operations that maybe performed to generate paraphrases of ground truth questions to traina question answering system in accordance with various embodiments;

FIG. 5 shows a flow diagram illustrating aspects of operations that maybe performed to generate paraphrases of ground truth questions to traina question answering system in accordance with various embodiments;

FIG. 6 shows a flow diagram illustrating aspects of operations that maybe performed to generate paraphrases of ground truth questions to traina question answering system in accordance with various embodiments; and

FIG. 7 shows an illustrative block diagram of an example data processingsystem that can be applied to implement embodiments of the presentdisclosure.

DETAILED DESCRIPTION

The quality of the responses provided by a QA system is tied to thetraining provided to the system. When a cognitive system is trained,ground truth is provided to the system. The quality of system training,and in turn, the quality of the cognitive system is determined by thequality of the ground truth used to train the system. Therefore, themore comprehensive the ground truth, the higher the quality of thesystem training. Therefore, it is desirable to develop a system toautomatically generate additional questions as a part of the groundtruth so that a training system may provide higher quality training.

FIG. 1 shows a block diagram of a system 100 that provides training of aQA system in accordance with various embodiments. The system 100includes a QA system 106 and a training system 102. The QA system 106 isa machine learning system that receives training from the trainingsystem 102. The training guides and adjusts the operation of the QAsystem 106 to improve the quality of the answers provided by the QAsystem 106. The QA system 106 is illustrative and is not intended tostate or imply any limitation with regard to the type of QA mechanismswith which various embodiments may be implemented. Many modifications tothe example QA system 100 may be implemented in various embodiments.

The system 100, including the QA system 106 and the training system 102may be implemented on one or more computing devices (comprising one ormore processors and one or more memories, and optionally including anyother computing device elements generally known in the art includingbuses, storage devices, communication interfaces, and the like).

The QA system 100 operates by accessing information from a corpus ofdata or information (also referred to as a corpus of content), analyzingit, and then generating answer results based on the analysis of thisdata. Accessing information from a corpus of data typically includes: adatabase query that answers questions about what is in a collection ofstructured records, and a search that delivers a collection of documentlinks in response to a query against a collection of unstructured data(text, markup language, etc.). Conventional question answering systemsare capable of generating answers based on the corpus of data and theinput question, verifying answers to a collection of questions for thecorpus of data, correcting errors in digital text using a corpus ofdata, and selecting answers to questions from a pool of potentialanswers, i.e. candidate answers.

The QA system 106 includes question processing 108, answer processing110, and databases 112. The databases 112 store documents 114 that serveas at least a part of the corpus of content from which answers toquestions are derived. The documents 114 may include any file, text,article, or source of data for use in the QA system 106. The questionprocessing 108 receives questions to be answered by the QA system 106.The questions may be formed using natural language. The questions may beprovided by the training system 102 to facilitate training of the QAsystem 106, or may be provided by users of the QA system 106. Thetraining system 102 may be coupled to the QA system 106 via a network,such as a local area network, a wide area network, the internet, orother communication system.

In some illustrative embodiments, the QA system 106 may be the IBMWatson™ QA system available from International Business MachinesCorporation of Armonk, N.Y. The IBM Watson™ QA system may receive aninput question which it then parses to extract the major features of thequestion, that in turn are then used to formulate queries that areapplied to the corpus of data. Based on the application of the queriesto the corpus of data, a set of hypotheses, or candidate answers to theinput question, are generated by looking across the corpus for portionsof the corpus of data that have some potential for containing a valuableresponse to the input question.

The IBM Watson™ QA system analyzes the language of the input questionand the language used in each of the portions of the corpus of datafound during the application of the queries using a variety of reasoningalgorithms. There may be hundreds or even thousands of reasoningalgorithms applied, each of which performs different analysis, e.g.,comparisons, and generates a score. For example, some reasoningalgorithms may look at the matching of terms and synonyms within thelanguage of the input question and the found portions of the corpus ofdata. Other reasoning algorithms may look at temporal or spatialfeatures in the language, while others may evaluate the source of theportion of the corpus of data and evaluate its veracity.

The scores obtained from the various reasoning algorithms indicate theextent to which the potential response is inferred by the input questionbased on the specific area of focus of that reasoning algorithm. Eachresulting score is then weighted against a statistical model. Thestatistical model captures how well the reasoning algorithm performed atestablishing the inference between two similar passages for a particulardomain during the training period of the IBM Watson™ QA system. Thestatistical model may then be used to summarize a level of confidencethat the IBM Watson™ QA system has regarding the evidence that thepotential response, i.e. candidate answer, is inferred by the question.This process may be repeated for each of the candidate answers until theIBM Watson™ QA system identifies candidate answers that surface as beingsignificantly stronger than others and thus, generates a final answer,or ranked set of answers, for the input question.

The question processing 108 receives input questions that are presentedin a natural language format. That is, a user of the training system 102may input, via a user interface, an input question to obtain an answer.In response to receiving the input question, the question processing 108parses the input question using natural language processing techniquesto extract major features from the input question, classify the majorfeatures according to types, e.g., names, dates, or any of a variety ofother defined topics. The identified major features may then be used todecompose the question into one or more queries that may be submitted tothe databases 112 in order to generate one or more hypotheses. Thequeries may be generated in any known or later developed query language,such as the Structure Query Language (SQL), or the like. The queries maybe submitted to one or more databases 112 storing the documents 114 andother information.

The queries may be submitted to one or more databases 112 storinginformation about the electronic texts, documents, articles, websites,and the like, that make up the corpus of data/information. The queriesare submitted to the databases 112 to generate results identifyingpotential hypotheses for answering the input question. That is, thesubmission of the queries results in the extraction of portions of thecorpus of data/information matching the criteria of the particularquery. These portions of the corpus are analyzed and used to generatehypotheses for answering the input question. These hypotheses are alsoreferred to herein as “candidate answers” for the input question. Forany input question, there may be hundreds of hypotheses or candidateanswers generated that need to be evaluated.

The answer processing 110 analyzes and compares the language of theinput question and the language of each hypothesis or “candidate answer”as well as performs evidence scoring to evaluate the likelihood that aparticular hypothesis is a correct answer for the input question. Asmentioned above, this process may involve using a plurality of reasoningalgorithms, each performing a separate type of analysis of the languageof the input question and/or content of the corpus that providesevidence in support of, or not, of the hypothesis. Each reasoningalgorithm generates a score based on the analysis it performs whichindicates a measure of relevance of the individual portions of thecorpus of data/information extracted by application of the queries aswell as a measure of the correctness of the corresponding hypothesis,i.e. a measure of confidence in the hypothesis.

The answer processing 110 may synthesize the large number of relevancescores generated by the various reasoning algorithms into confidencescores for the various hypotheses. This process may involve applyingweights to the various scores, where the weights have been determinedthrough training of the statistical model employed by the QA system 106.The weighted scores may be processed in accordance with a statisticalmodel generated through training of the QA system 106 that identifies amanner by which these scores may be combined to generate a confidencescore or measure for the individual hypotheses or candidate answers.This confidence score or measure summarizes the level of confidence thatthe QA system 106 has about the evidence that the candidate answer isinferred by the input question, i.e. that the candidate answer is thecorrect answer for the input question.

In the answer processing 110, the resulting confidence scores ormeasures may be compared against predetermined thresholds, or otheranalysis may be performed on the confidence scores to determine whichhypotheses/candidate answers are most likely to be the answer to theinput question. The hypotheses/candidate answers may be ranked accordingto these comparisons to generate a ranked listing ofhypotheses/candidate answers (hereafter simply referred to as “candidateanswers”). From the ranked listing of candidate answers, a final answerand confidence score, or final set of candidate answers and confidencescores, may be generated and output to the submitter of the originalinput question.

The training system 102 facilitates configuration of the QA system 106to provide answers to submitted questions and to improve the quality ofthe answers provided to submitted questions. The quality of the answersto a submitted question may be improved by selecting the candidateanswers that are most relevant to the question. The quality of theanswers provided by the QA system 106 is related to the ground truththat is used to train the QA system 106. Embodiments of the trainingsystem 102 improve the quality of the answers provided by the QA system106 by enriching the ground truth that is used to train the QA system106. Ground truth is questions, in the form of phrases and/or sentences,which are mapped to a known intent and/or answer. For example, thequestions, “How can I close my account?” and “Is there a way I can shutmy account?” may be mapped to an intent “closing an account.” In anotherexample, the questions, “What is the capital of California?” and “Whatis the capital of CA?” may be mapped to the answer “Sacramento.” The QAsystem 106 then may be trained to determine that similar questions tothe questions provided as part of the ground truth should providesimilar answers because the intent of the questions is the same.

In one embodiment, the training system 102 provides a number ofquestions to the QA system 106. The questions provided by the trainingsystem 102 to the QA system 106 are referred to herein as “trainingquestions.” The QA system 106 processes the training questions andselects a number of candidate answers for each of the trainingquestions. The QA system 106 provides the candidate answers to thetraining system 102. The candidate answers generated by the QA system106 are compared by the training system 102 to correct answers in theground truth. That is, the training performed by training system 102 mayinclude the use of a known input question on a known training set ofdata with the goal being for the QA system 106 to generate the knowncorrect answers found in the ground truth. By comparing the candidateanswers to the known correct answers in the ground truth using logicfound in the training system 102, the training system 102 may determinewhether the QA system 106 is operating in the desired manner and wheredifferences occur between the answers generated by the QA system 106 andthe correct answers. In the event that the QA system 106 returns anincorrect answer to a training question, the training system 102 mayadjust the logic and/or algorithms of the QA system 106, and moreparticularly, the answer processing 110 to decrease the confidence scorefor the incorrectly provided answer. In the event that the QA system 106returns a correct answer to the training question, the training system102 may adjust the logic and/or algorithms of the QA system 106, andmore particularly, the answer processing 110 to increase the confidencescore for the correctly provided answer. In this way, the trainingsystem 102 is able to train the QA system 106 to provide correct answersto input questions.

In some embodiments, the ground truth is provided to the training system102 by customers, experts, and/or generated by the training systemitself through, for example, crawling encyclopedias for questions andanswers. In an embodiment, the training system 102 receives sparseground truth 116 as the initial ground truth for use by the trainingsystem. Sparse ground truth 116 is ground truth for a particular intentthat includes a limited number of questions that are mapped to thatintent. For example, sparse ground truth 116 may include only twoquestions mapped to a single intent. Thus, the sparse ground truth 116may provide lesser training capabilities for training system 102 thanmore enriched ground truth (i.e., ground truth with more questionsmapped to the single intent). Because the questions contained in thesparse ground truth 116 are mapped to the same intent, they may beconsidered paraphrases of one another.

In order to provide a more robust training system 102, paraphrasegeneration system 104 is configured to receive the sparse ground truth116 and generate additional paraphrases to increase the ground truthutilized by training system 102 to train QA system 106. For example, thetraining system 102 may receive only two questions, “How can I close myaccount?” and “Is there a way I can shut my account?” that are mapped tothe intent “closing an account” as part of the sparse ground truth 116.The paraphrase generation system 104 is configured to generateparaphrases of these two questions, and thus, increase the ground truthutilized to train QA system 106. For example, the paraphrase generationsystem 104 may generate questions such as, “Is there a way for me toclose my account?”, “How can I shut my account?”, “How could I close myaccount?”, etc. as additional questions mapped to the intent “closing anaccount.” Thus, the ground truth is automatically enriched by thetraining system 102 through the generation of additional paraphrases ofthe original questions in the sparse ground truth 116.

FIG. 2 shows an illustrative block diagram of training system 102 thatincludes paraphrase generation system 104 to generate paraphrases ofground truth questions to provide training of a QA system 106 inaccordance with various embodiments. The paraphrase generation systemmay include a logical form (LF) generation system 204, a phrasal editgeneration system 206, a disjunctive logical form (DLF) generationsystem 208, surface realizer 210, and pruning system 212. As discussedabove, the paraphrase generation system 104 receives the sparse groundtruth 116. In other words, paraphrase generation system 104 receives atleast a first phrase and/or sentence and a second phrase and/or sentence(i.e., questions) mapped to a single intent. Because the first questionand the second question are mapped to the same intent, the firstquestion and the second question are paraphrases of each other. Forexample, the first question may be, “How can I close my account?” whilethe second question may be, “Is there a way I can shut my account?”

The LF generation system 204 is configured to receive the first andsecond questions and convert the first and second questions into logicalforms. In other words, the LF generation system 204 is configured toreceive the first question and convert it into a first logical form andreceive the second question and convert it into a second logical form. Alogical form of a sentence is the form of the sentence after the subjectmatter of the sentences content terms have been abstracted. In otherwords, the content terms of sentence may be regarded as blanks on aform. In some embodiments, the LF generation system 204 may create thelogical forms that result from a statistical parse of the first andsecond questions. For example, the LF generation system 204 maydependency parse the two questions to generate a parsed representationof each of the questions. Dependency parsing establishes relationshipsbetween specific words in the question and words which modify thosespecific words. In other words, dependency parsing allows the LFgeneration system 204 to identify the component parts of speech (e.g.,subject, adjective, verb, adverb, object, etc.) of the words in the twoquestions.

The phrasal edit generation system 206 receives the logical forms of thefirst and second questions, including the parsed representation of eachquestion, and generates a number of phrasal edits. The phrasal editsinclude differences between the logical forms of the two sentences. Inother words, the phrasal edit generation system 206 receives the parsedrepresentations of each question and determines differences between thetwo representations, both syntactically and semantically. For example,the phrasal edit generation system 206 may determine that in thequestion “Is it a bright day?” the term “bright” corresponds with theterm “sunny” in the question “Is it a sunny day?” because both “bright”and “sunny” are adjectives that modify “day.” Thus, the phrasal editgeneration system 206 may generate a variety of phrases from the twoquestions which are edits of one another.

The DLF generation system 208 is configured to receive the phrasal editsfrom the phrasal edits generation system 206 and convert the phrasaledits into disjunctive logical forms in two directions. In other words,the DLF generation system 208 generates bidirectional disjunctivelogical forms. For example, in the English language, sentences aregenerally read from left to right. Thus, humans typically assume thatthe structure of the sentence is from left to right only. However, thesentence also has structure from right to left. To a machine, there isno difference between a left to right structure or a right to leftstructure. Thus, the DLF generation system 208 generates disjunctivelogical forms in both directions (i.e., from left to right and fromright to left).

Each of the bidirectional disjunctive logical forms includesdisjunctions that represent alternative choices of words at a level ofsemantic dependencies. A disjunction may be a syntactic construction inwhich two or more words of the same component part of speech arecombined while maintaining the same semantic relations with surroundingelements of the sentence. In some embodiments, to generate thebidirectional disjunctive logical forms, the DLF generation system 208may generate bigram models from the phrasal edits. A bigram model is asequence of two adjacent words with a frequency distribution of thosetwo adjacent words. Thus, knowing one word (the first word) in thebigram allows the paraphrase generation system 202 to predict the nextword (the second word). Because the disjunctive logical forms arebidirectional, knowing one word (the second word) in the bigram allowsthe paraphrase generation system 202 to predict the previous word (thefirst word) as well. In this way, the DLF generation system 208 mayconstruct a trigram model. A trigram model is a sequence of threeadjacent words with a frequency distribution of the adjacent words.Thus, knowing one word allows the DLF generation system 208 to predictboth the previous word and the next word. Hence, the bidirectionality ofthe disjunctive logical forms allows the paraphrase generation system202 to introduce greater variation in the number of question paraphrasesthat may be generated because the structure and language of the firstand second questions are analyzed from both left to right and from rightto left. In other words, utilizing bidirectional disjunctive logicalforms enables the examination of natural language for a richerunderstanding of syntactic construct thereby resulting in the creationof more refined and accurate syntactic variants that retain semanticmeaning of the original two questions.

Surface realizer 210 is configured to receive and realize thebidirectional disjunctive logical forms to generate paraphrases of thefirst and second questions. For example, surface realizer 210 may be anysurface realizer that takes in the bidirectional disjunctive logicalforms and produces an ordered sequence of words that are constrained bygrammatical rules and lexicon. In other words, the surface realizer 210takes the syntactic representation of text from the bidirectionaldisjunctive logical forms and generates natural language text. Due tothe variations created by the DLF generation system 208 from thebidirectional analysis in generating the disjunctive logical forms,lexical diversity is increased. In an example, the surface realizer 210utilizes OpenCCG to generate the paraphrases.

Pruning system 212 is configured to receive and, in some embodiments,score the paraphrases generated by the surface realizer 210. Pruningsystem 212 may score each of the paraphrases generated by the surfacerealizer 210 based on textual similarity between the first and secondquestions of the sparse truth 116 and each respective paraphrasegenerated by the surface realizer 210. In some embodiments, the pruningsystem 212 may compare a first paraphrase generated by the surfacerealizer 210 with the first and second questions from the sparse groundtruth 116 and generate a score based on the syntactic variation andsemantic meaning of the paraphrase generated by the surface realizer210. A paraphrase generated by surface realizer 210 may be assigned ahigher score if the syntactic variation with respect to the first andsecond questions is larger than if the syntactic variation is lower, solong as the semantic meaning remains similar to the semantic meaning ofthe first and second questions. In other words, the pruning system 212generates scores that bias towards paraphrases that bear highersyntactic variation while maintaining semantic meaning and bias againstparaphrases that bear minimal syntactic variation while maintainingsemantic meaning.

Pruning system 212 then may prune the paraphrases generated by thesurface realizer 210 to a list of n-best paraphrases that providegrammatical alternatives to the original first and second questions aswell as paraphrases that mix and match content (e.g., words) across thepair of questions. In some embodiments, the pruning system 212 mayremove the paraphrases generated by the surface realizer 210 with thelowest scores so that the n-best paraphrases are maintained. Forexample, if the surface realizer 210 generates 30 paraphrases for thefirst and second questions, the pruning system 212 may remove the lowestscoring 22-24 so that 6-8 paraphrases remain in the list of n-bestparaphrases. In other examples, any number of paraphrases may be prunedfrom the paraphrases generated by surface realizer 210. In this way, theparaphrase generation system 202 may generate additional questions toenrich the ground truth utilized to train the QA system 106.

The enriched ground truth, including the original two questions from thesparse ground truth 116 and the generated n-best paraphrases which aremapped to an intent, may be provided to trainer 214 for training the QAsystem 106. In some embodiments, the enriched ground truth is stored ina storage media (i.e., memory) in the trainer 214. The trainer 214 maytrain the QA system as discussed above. For example, the trainer 214 mayreceive results of a known input question (i.e., a training question)from the QA system 216. By comparing the results to the known correctanswers in the ground truth using logic found in the trainer 214, thetrainer 214 may determine whether the QA system 106 is operating in thedesired manner and where differences occur between the answers generatedby the QA system 106 and the correct answers. In the event that the QAsystem 106 returns an incorrect answer to a training question, thetrainer 214 may adjust the logic and/or algorithms of the QA system 106,and more particularly, the answer processing 110 to decrease theconfidence score for the incorrectly provided answer. In the event thatthe QA system 106 returns a correct answer to the training question, thetrainer 214 may adjust the logic and/or algorithms of the QA system 106,and more particularly, the answer processing 110 to increase theconfidence score for the correctly provided answer. In this way, thetraining system 102 is able to train the QA system 106 to providecorrect answers to input questions.

FIG. 3 shows a flow diagram illustrating aspects of operations that maybe performed to generate paraphrases of ground truth questions to traina question answering system in accordance with various embodiments.Though depicted sequentially as a matter of convenience, at least someof the actions shown can be performed in a different order and/orperformed in parallel. Additionally, some embodiments may perform onlysome of the actions shown. In some embodiments, at least some of theoperations of the method 300 may be provided by instructions executed bya computer of the system 100.

The method 300 begins in block 302 with receiving a first and a secondphrase, such as a question and/or other sentence. The first and secondphrases may be a part of a sparse ground truth, such as sparse groundtruth 116 and may be created by the training system 102, by anothercomputer system, and/or by a human. In block 304, the method 300continues with converting the first and second phrases into logicalforms. For example, the first phrase may be dependency parsed toidentify parts of speech of the first phrase thereby generating a firstparsed representation of the first phrase. Similarly, the second phrasemay be dependency parsed to identify parts of speech of the secondphrase thereby generating a second parsed representation of the secondphrase.

The method 300 continues in block 306 with generating phrasal edits fromthe logical forms. For example, the first parsed representation of thefirst phrase may be compared with the second parsed representation ofthe second phrase to determine one or more words from the first phrasethat correspond with one or more words from the second phrase. In otherwords, a variety of phrases from the first and second phrases which areedits of one another may be identified. In an example, the phrasal editgeneration system 206 may determine that in the question “Is it a brightday?” the term “bright” corresponds with the term “sunny” in thequestion “Is it a sunny day?” because both “bright” and “sunny” areadjectives that modify “day.”

In block 308, the method 300 continues with converting the phrasal editsinto bidirectional disjunctive logical forms. Each of the bidirectionaldisjunctive logical forms may include disjunctions that representalternative choices of words at a level of semantic dependencies. Insome embodiments bigram models may be generated from the phrasal edits.Thus, knowing one word (the first word) in the bigram allows aprediction of the next word (the second word) and, because thedisjunctive logical forms are bidirectional, knowing one word (thesecond word) in the bigram allows a prediction of the previous word (thefirst word). Hence, a trigram model, which is a bidirectional frequencydistribution model for the words in the first and second phrases, may begenerated such that knowing one word allows a prediction of both theprevious word and the next word.

The method 300 continues in block 310 with generating a first pluralityof paraphrases. For example, the bidirectional disjunctive logical formsmay be fed into a surface realizer, such as surface realizer 210, whichgenerates a number of ordered sequences of words that are constrained bygrammatical rules and lexicon. In block 312, the method 300 continueswith pruning the first plurality of paraphrases to yield a secondplurality of paraphrases containing grammatical alternatives to thefirst and second phrases. For example, scores for each of the firstplurality of paraphrases may be generated based on the syntacticvariation of the each respective paraphrase with the first and secondphrases. Paraphrases from the first plurality of paraphrases with thelowest scores may be removed so that an n-best number of paraphrasesremain. For example, if 30 paraphrases are generated as part of thefirst plurality of paraphrases, the lowest scoring 22-24 paraphraseswithin the first plurality of paraphrases may be removed so that 6-8paraphrases remain in the list of n-best paraphrases.

FIG. 4 shows a flow diagram illustrating aspects of operations that maybe performed to generate paraphrases of ground truth questions to traina question answering system in accordance with various embodiments.Though depicted sequentially as a matter of convenience, at least someof the actions shown can be performed in a different order and/orperformed in parallel. Additionally, some embodiments may perform onlysome of the actions shown. In some embodiments, at least some of theoperations of the method 400 may be provided by instructions executed bya computer of the system 100.

The method 400 begins in block 402 with receiving a first and a secondphrase, such as a question and/or other sentence. The first and secondphrases may be a part of a sparse ground truth, such as sparse groundtruth 116 and may be created by the training system 102, by anothercomputer system, and/or by a human. In block 404, the method 400continues with dependency parsing the first and second phrases toidentify parts of speech of the first and second phrases therebygenerating a first parsed representation of the first phrase and asecond parsed representation of the second phrase.

The method 400 continues in block 406 with comparing the first parsedrepresentation of the first phrase with the second parsed representationof the second phrase to determine one or more words from the firstphrase that correspond with a word from the second phrase. In otherwords, a variety of phrases from the first and second phrases which areedits of one another may be identified. In an example, the phrasal editgeneration system 206 may determine that in the question “Is it a brightday?” the term “bright” corresponds with the term “sunny” in thequestion “Is it a sunny day?” because both “bright” and “sunny” areadjectives that modify “day.”

In block 408, the method 400 continues with generating one or moretrigram models. A trigram model, which is a bidirectional frequencydistribution model for the words in the first and second phrases, may begenerated such that knowing one word allows a prediction of both theprevious word and the next word. The method 400 continues in block 410with generating a first plurality of paraphrases. For example, a surfacerealizer 210 may generate a number of ordered sequences of words thatare constrained by grammatical rules and lexicon based on the one ormore trigram models. In block 412, the method 400 continues with pruningthe first plurality of paraphrases to yield a second plurality ofparaphrases containing grammatical alternatives to the first and secondphrases. For example, scores for each of the first plurality ofparaphrases may be generated based on the syntactic variation of theeach respective paraphrase with the first and second phrases.Paraphrases from the first plurality of paraphrases with the lowestscores may be removed so that an n-best number of paraphrases remain.For example, if 30 paraphrases are generated as part of the firstplurality of paraphrases, the lowest scoring 22-24 paraphrases withinthe first plurality of paraphrases may be removed so that 6-8paraphrases remain in the list of n-best paraphrases.

FIG. 5 shows a flow diagram illustrating aspects of operations that maybe performed to generate paraphrases of ground truth questions to traina question answering system in accordance with various embodiments.Though depicted sequentially as a matter of convenience, at least someof the actions shown can be performed in a different order and/orperformed in parallel. Additionally, some embodiments may perform onlysome of the actions shown. In some embodiments, at least some of theoperations of the method 500 may be provided by instructions executed bya computer of the system 100.

The method 500 begins in block 502 with receiving bidirectionaldisjunctive logical forms, in some embodiments by a surface realizer,such as surface realizer 210. The bidirectional disjunctive logicalforms include two direction disjunctions (i.e., bidirectional) ofdifferences between a first logical form of a first sentence and asecond logical form of a second sentence. For example, each of thebidirectional disjunctive logical forms may include disjunctions thatrepresent alternative choices of words at a level of semanticdependencies.

The method 500 continues in block 504 with realizing the bidirectionaldisjunctive logical forms to generate a first plurality of paraphrasesof the first and second sentences. For example, the bidirectionaldisjunctive logical forms may be fed into a surface realizer, such assurface realizer 210, which generates a number of ordered sequences ofwords that are constrained by grammatical rules and lexicon. In block506, the method 500 continues with scoring each of the first pluralityof paraphrases based on textual similarity between the first pluralityof paraphrases and the first and second sentences. For example, aparaphrase may be assigned a higher score if the syntactic variationwith respect to the first and second sentences is larger than if thesyntactic variation is lower, so long as the semantic meaning remainssimilar to the semantic meaning of the first and second sentences. Inother words, scores may be biased such that paraphrases that bear highersyntactic variation while maintaining semantic meaning receive higherscores and paraphrases that bear minimal syntactic variation whilemaintaining semantic meaning receive lower scores.

The method 500 continues in block 508 with pruning, based on the scoresof each of the first plurality of paraphrases, the first plurality ofparaphrases to generate a second plurality of paraphrases. For example,paraphrases from the first plurality of paraphrases with the lowestscores may be removed so that an n-best number of paraphrases remain.Thus, if 30 paraphrases are generated as part of the first plurality ofparaphrases, the lowest scoring 22-24 paraphrases within the firstplurality of paraphrases may be removed so that 6-8 paraphrases remainin the list of n-best paraphrases.

In block 510, the method 500 continues with providing the secondplurality of paraphrases (i.e., the n-best paraphrases) to a trainer fortraining a QA system. For example, the second plurality of paraphrasesmay be provided to a trainer, such as trainer 214 as part of enrichedground truth that may be utilized to train QA system 106. The method 500continues in block 512 with training the QA system. For example, thetrainer may compare the results of a training query of the QA system tothe known correct answers in the enriched ground truth. In the eventthat the QA system returns an incorrect answer to a training question,the trainer may adjust the QA system to decrease a confidence score forthe incorrectly provided answer. In the event that the QA system returnsa correct answer to the training question, the trainer may adjust the QAsystem to increase the confidence score for the correctly providedanswer.

FIG. 6 shows a flow diagram illustrating aspects of operations that maybe performed to generate paraphrases of ground truth questions to traina question answering system in accordance with various embodiments.Though depicted sequentially as a matter of convenience, at least someof the actions shown can be performed in a different order and/orperformed in parallel. Additionally, some embodiments may perform onlysome of the actions shown. In some embodiments, at least some of theoperations of the method 600 may be provided by instructions executed bya computer of the system 100.

The method 600 begins in block 602 with dependency parsing the first andsecond phrases to identify parts of speech of the first and secondphrases thereby generating a first parsed representation of the firstphrase and a second parsed representation of the second phrase. In block604, the method 600 continues with comparing the first parsedrepresentation of the first phrase with the second parsed representationof the second phrase to determine one or more words from the firstphrase that correspond with a word from the second phrase. In otherwords, a variety of phrases from the first and second phrases which areedits of one another may be identified. In an example, the phrasal editgeneration system 206 may determine that in the question “Is it a brightday?” the term “bright” corresponds with the term “sunny” in thequestion “Is it a sunny day?” because both “bright” and “sunny” areadjectives that modify “day.”

The method 600 continues in block 606 with realizing the bidirectionaldisjunctive logical forms to generate a first plurality of paraphrasesof the first and second sentences. For example, a surface realizer, suchas surface realizer 210, may generate a number of ordered sequences ofwords that are constrained by grammatical rules and lexicon based on thecomparison of the first and second parsed representations. In block 608,the method 600 continues with scoring each of the first plurality ofparaphrases based on textual similarity between the first plurality ofparaphrases and the first and second sentences. For example, aparaphrase may be assigned a higher score if the syntactic variationwith respect to the first and second sentences is larger than if thesyntactic variation is lower, so long as the semantic meaning remainssimilar to the semantic meaning of the first and second sentences. Inother words, scores may be biased such that paraphrases that bear highersyntactic variation while maintaining semantic meaning receive higherscores and paraphrases that bear minimal syntactic variation whilemaintaining semantic meaning receive lower scores.

The method 600 continues in block 610 with pruning, based on the scoresof each of the first plurality of paraphrases, the first plurality ofparaphrases to generate a second plurality of paraphrases. For example,paraphrases from the first plurality of paraphrases with the lowestscores may be removed so that an n-best number of paraphrases remain.Thus, if 30 paraphrases are generated as part of the first plurality ofparaphrases, the lowest scoring 22-24 paraphrases within the firstplurality of paraphrases may be removed so that 6-8 paraphrases remainin the list of n-best paraphrases.

In block 612, the method 600 continues with providing the secondplurality of paraphrases (i.e., the n-best paraphrases) to a trainer fortraining a QA system. For example, the second plurality of paraphrasesmay be provided to a trainer, such as trainer 214 as part of enrichedground truth that may be utilized to train QA system 106. The method 600continues in block 614 with training the QA system. For example, thetrainer may compare the results of a training query of the QA system tothe known correct answers in the enriched ground truth. In the eventthat the QA system returns an incorrect answer to a training question,the trainer may adjust the QA system to decrease a confidence score forthe incorrectly provided answer. In the event that the QA system returnsa correct answer to the training question, the trainer may adjust the QAsystem to increase the confidence score for the correctly providedanswer.

FIG. 7 is a block diagram of an example data processing system in whichaspects of the illustrative embodiments may be implemented. Dataprocessing system 700 is an example of a computer that can be applied toimplement the training system 102, the QA system 106, or devicesproviding the sparse ground truth 116 to the training system 102 in FIG.1, in which computer usable code or instructions implementing theprocesses for illustrative embodiments of the present invention may belocated. In one illustrative embodiment, FIG. 7 represents a computingdevice that implements the training system 102 augmented to include theadditional mechanisms of the illustrative embodiments describedhereafter.

In the depicted example, data processing system 700 employs a hubarchitecture including north bridge and memory controller hub (NB/MCH)706 and south bridge and input/output (I/O) controller hub (SB/ICH) 710.Processor(s) 702, main memory 704, and graphics processor 708 areconnected to NB/MCH 706. Graphics processor 708 may be connected toNB/MCH 706 through an accelerated graphics port (AGP).

In the depicted example, local area network (LAN) adapter 716 connectsto SB/ICH 710. Audio adapter 730, keyboard and mouse adapter 722, modem724, read only memory (ROM) 726, hard disk drive (HDD) 712, CD-ROM drive714, universal serial bus (USB) ports and other communication ports 718,and PCI/PCle devices 720 connect to SB/ICH 710 through bus 732 and bus734. PCI/PCle devices may include, for example, Ethernet adapters,add-in cards, and PC cards for notebook computers. PCI uses a card buscontroller, while PCle does not. ROM 726 may be, for example, a flashbasic input/output system (BIOS).

HDD 712 and CD-ROM drive 714 connect to SB/ICH 710 through bus 734. HDD712 and CD-ROM drive 714 may use, for example, an integrated driveelectronics (IDE) or serial advanced technology attachment (SATA)interface. Super I/O (SIO) device 728 may be connected to SB/ICH 710.

An operating system runs on processor(s) 702. The operating systemcoordinates and provides control of various components within the dataprocessing system 700 in FIG. 7. In some embodiments, the operatingsystem may be a commercially available operating system such asMicrosoft® Windows 10®. An object-oriented programming system, such asthe Java™ programming system, may run in conjunction with the operatingsystem and provides calls to the operating system from Java™ programs orapplications executing on data processing system 700.

In some embodiments, data processing system 700 may be, for example, anIBM® eServer™ System p® computer system, running the AdvancedInteractive Executive (AIX®) operating system or the LINUX® operatingsystem. Data processing system 700 may be a symmetric multiprocessor(SMP) system including a plurality of processors 702. Alternatively, asingle processor system may be employed.

Instructions for the operating system, the object-oriented programmingsystem, and applications or programs are located on storage devices,such as HDD 712, and may be loaded into main memory 704 for execution byprocessor(s) 702. The processes for illustrative embodiments of thepresent invention may be performed by processor(s) 702 using computerusable program code, which may be located in a memory such as, forexample, main memory 704, ROM 726, or in one or more peripheral devices712 and 714, for example.

A bus system, such as bus 732 or bus 734 as shown in FIG. 7, may includeone or more buses. The bus system may be implemented using any type ofcommunication fabric or architecture that provides for a transfer ofdata between different components or devices attached to the fabric orarchitecture. A communication unit, such as modem 724 or network adapter716 of FIG. 7, may include one or more devices used to transmit andreceive data. A memory may be, for example, main memory 704, ROM 726, ora cache such as found in NB/MCH 706 in FIG. 7.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or eternal storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions, instruction-setarchitecture (ISA) instructions, machine instructions, machine dependentinstructions, microcode, firmware instructions, state-setting data,configuration data for integrated circuitry, or either source code orobject code written in any combination of one or more programminglanguages, including an object oriented programming language such asSmalltalk, C++, or the like, and procedural programming languages, suchas the “C” programming language or similar programming languages. Thecomputer readable program instructions may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider). In some embodiments, electronic circuitry including, forexample, programmable logic circuitry, field-programmable gate arrays(FPGA), or programmable logic arrays (PLA) may execute the computerreadable program instructions by utilizing state information of thecomputer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

1. A method, comprising: receiving, by a surface realizer of aparaphrase generation system, a plurality of bidirectional disjunctivelogical forms, wherein the plurality of bidirectional disjunctivelogical forms includes two directional disjunctions of differencesbetween a first logical form of a first sentence and a second logicalform of a second sentence; realizing, by the surface realizer, theplurality of bidirectional disjunctive logical forms to generate firstparaphrases of the first and second sentences; determining a first scorefor a third paraphrase of the first paraphrases; determining a secondscore for a fourth paraphrase of the first paraphrases, wherein thefirst score is higher than the second score based in part on a firstsyntactic variation between the third paraphrase and the first sentenceand the second sentence being greater than a second syntactic variationbetween the fourth paraphrase and the first sentence and the secondsentence; and pruning, by the paraphrase generation system, the firstparaphrases to generate second paraphrases, wherein the firstparaphrases are pruned based on the first score and the second scoresuch that the third paraphrase is included in the second paraphrases andthe fourth paraphrase is not included in the second paraphrases based onthe first score being higher than the second score.
 2. The method ofclaim 1, wherein the differences include a first word that is analternative to a second word and has a similar meaning to the secondword, and wherein the two directional disjunctions include the firstword combined with the second word while maintaining the same semanticrelations with surrounding elements of a third sentence.
 3. The methodof claim 1, wherein realizing the plurality of bidirectional disjunctivelogical forms includes generating natural language text from syntacticrepresentations of text from the plurality of bidirectional disjunctivelogical forms.
 4. The method of claim 1, further comprising: converting,by the paraphrase generation system, the first sentence into the firstlogical form by dependency parsing the first sentence to identify partsof speech of the first sentence thereby generating a first parsedrepresentation of the first sentence; and converting, by the paraphrasegeneration system, the second sentence into the second logical form bydependency parsing the second sentence to identify parts of speech ofthe second sentence thereby generating a second parsed representation ofthe second sentence.
 5. The method of claim 4, further comprisinggenerating, by the paraphrase generation system, at least one of theplurality of bidirectional disjunctive logical forms by comparing thefirst parsed representation of the first sentence with the second parsedrepresentation of the second sentence to determine a word from the firstsentence that corresponds with a word from the second sentence, andwherein realizing the plurality of bidirectional disjunctive logicalforms includes generating natural language text from syntacticrepresentations of text from the plurality of bidirectional disjunctivelogical forms.
 6. The method of claim 1, further comprising, providingthe second of paraphrases to a training system as ground truth fortraining.
 7. The method of claim 6, further comprising, training aquestion answering system utilizing the ground truth.
 8. A system,comprising: a processor; and a memory coupled to the processor, thememory encoded with instructions that when executed by the processorcause the processor to: receive a plurality of bidirectional disjunctivelogical forms, wherein the plurality of bidirectional disjunctivelogical forms includes two directional disjunctions of differencesbetween a first logical form of a first sentence and a second logicalform of a second sentence; realize the plurality of bidirectionaldisjunctive logical forms to generate first paraphrases of the first andsecond sentences; determine a first score for a third paraphrase of thefirst paraphrases; determine a second score for a fourth paraphrase ofthe first paraphrases, wherein the first score is higher than the secondscore based in part on a first syntactic variation between the thirdparaphrase and the first sentence and the second sentence being greaterthan a second syntactic variation between the fourth paraphrase and thefirst sentence and the second sentence; and prune the first paraphrasesto generate second paraphrases, wherein the first paraphrases are prunedbased on the first score and the second score such that the thirdparaphrase is included in the second paraphrases and the fourthparaphrase is not included in the second paraphrases based on the firstscore being higher than the second score.
 9. The system of claim 8,wherein the differences include a first word that is an alternative to asecond word and has a similar meaning to the second word, wherein thetwo directional disjunctions include the first word combined with thesecond word.
 10. The system of claim 8, wherein realizing the pluralityof bidirectional disjunctive logical forms includes generating naturallanguage text from syntactic representations of text from the pluralityof bidirectional disjunctive logical forms.
 11. The system of claim 8,wherein the instructions, when executed by the processor, further causethe processor to: convert the first sentence into the first logical formby dependency parsing the first sentence to identify parts of speech ofthe first sentence thereby generating a first parsed representation ofthe first sentence; and convert the second sentence into the secondlogical form by dependency parsing the second sentence to identify partsof speech of the second sentence thereby generating a second parsedrepresentation of the second sentence.
 12. The system of claim 11,wherein the instructions, when executed by the processor, further causethe processor to generate at least one of the plurality of bidirectionaldisjunctive logical forms by comparing the first parsed representationof the first sentence with the second parsed representation of thesecond sentence to determine a word from the first sentence thatcorresponds with a word from the second sentence.
 13. The system ofclaim 8, wherein the processor is configured to cause the memory tostore the second paraphrases as ground truth for training.
 14. Thesystem of claim 13, further comprising a training system coupled to theprocessor, wherein the training system is configured to train a questionanswering system utilizing the ground truth.
 15. A computer programproduct for generating paraphrases, the computer program productcomprising a computer readable storage medium having programinstructions embodied therewith, the program instructions executable bya computer to cause the computer to: receive a plurality ofbidirectional disjunctive logical forms, wherein the plurality ofbidirectional disjunctive logical forms includes two directionaldisjunctions of differences between a first logical form of a firstsentence and a second logical form of a second sentence; realize theplurality of bidirectional disjunctive logical forms to generate a firstplurality of paraphrases of the first and second sentences; determine afirst score for a third paraphrase of the first paraphrases; determine asecond score for a fourth paraphrase of the first paraphrases, whereinthe first score is higher than the second score based in part on a firstsyntactic variation between the third paraphrase and the first sentenceand the second sentence being greater than a second syntactic variationbetween the fourth paraphrase and the first sentence and the secondsentence; and prune the first paraphrases to generate secondparaphrases, wherein the first paraphrases are pruned based on the firstscore and the second score such that the third paraphrase is included inthe second paraphrases and the fourth paraphrase is not included in thesecond paraphrases based on the first score being higher than the secondscore.
 16. The computer program product of claim 15, wherein thedifferences include a first word that is an alternative to a second wordand has a similar meaning to the second word, and wherein the twodirectional disjunctions include the first word combined with the secondword.
 17. The computer program product of claim 15, wherein realizingthe plurality of bidirectional disjunctive logical forms includesgenerating natural language text from syntactic representations of textfrom the plurality of bidirectional disjunctive logical forms.
 18. Thecomputer program product of claim 15, wherein the program instructionsare further executable by the computer to cause the computer to: convertthe first sentence into the first logical form by dependency parsing thefirst sentence to identify parts of speech of the first sentence therebygenerating a first parsed representation of the first sentence; andconvert the second sentence into the second logical form by dependencyparsing the second sentence to identify parts of speech of the secondsentence thereby generating a second parsed representation of the secondsentence.
 19. The computer program product of claim 18, wherein theprogram instructions are further executable by the computer to cause thecomputer to generate at least one of the plurality of bidirectionaldisjunctive logical forms by comparing the first parsed representationof the first sentence with the second parsed representation of thesecond sentence to determine a word from the first sentence thatcorresponds with a word from the second sentence.
 20. The computerprogram product of claim 15, wherein the program instructions arefurther executable by the computer to cause the computer to store thesecond plurality of paraphrases as ground truth for training.